Move co-located containers onto shared foreman-core-network by Alleny244 · Pull Request #728 · theforeman/foremanctl

Alleny244 · 2026-08-05T13:53:04Z

Problem Statement

Embedded PostgreSQL defaults to listen_addresses = '*', exposing port 5432 on public interfaces when containers use host networking.

https://redhat.atlassian.net/browse/SAT-48503

Changes

Add a shared Podman bridge network foreman-core-network (10.130.0.0/24) via the foreman_core_network role, created early in deploy.
Move co-located containers off host networking onto that network (PostgreSQL, Valkey, Candlepin, Pulp, Foreman/Dynflow, IOP services).
Reach services by container name on the shared network (e.g. postgresql, valkey, candlepin, iop-core-gateway).
Publish host access on loopback only where needed (e.g. 127.0.0.1:5432, 127.0.0.1:6379, Candlepin/IOP gateway ports) so ports are not exposed on 0.0.0.0 / public interfaces.
Remove the IOP-only iop_network / iop-core-network; IOP uses foreman-core-network.
Update TLS SANs and Smart Proxy URL so Foreman can reach Candlepin and the IOP gateway by container name.
Leave foreman_proxy on host networking (edge/DHCP/DNS/TFTP/REX).
Add/update tests to ensure PostgreSQL (and Valkey) are not published on 0.0.0.0:5432 / [::]:5432 (and equivalent for Valkey).

Tests

./forge test --pytest-args="tests/postgresql_test.py -vv"

Checklist

Tests added/updated
Documentation updated (docs/iop.md)

ehelms · 2026-08-07T01:58:19Z

Here's two other ideas:

Put all the containers on the same network.

The iop network was created back in the foreman-installer for isolation, to reduce the need for TLS everywhere and because we were learning. Then we copied the idea here to start with parity. However, back when we implemented iop we always said this should get easier to manage once everything is in containers because we can normalize.

Put postgresql on both networks and have it listen to both.

Alleny244 · 2026-08-07T06:18:58Z

Here's two other ideas:

Put all the containers on the same network.

The iop network was created back in the foreman-installer for isolation, to reduce the need for TLS everywhere and because we were learning. Then we copied the idea here to start with parity. However, back when we implemented iop we always said this should get easier to manage once everything is in containers because we can normalize.

Put postgresql on both networks and have it listen to both.

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

ehelms · 2026-08-07T13:30:58Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Alleny244 · 2026-08-07T13:44:23Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Yeah, makes sense. I’ll look into the single network approach and see how we can handle the startup ordering.

Alleny244 · 2026-08-10T08:12:44Z

I’ll go with PostgreSQL on both networks here. Putting all containers on the same network would require broader changes, so a separate PR would be right?

Yes and no, you are ultimately trying to solve this broader problem of running only what's needed on the public interface. I'd rather us solve the architecture question than patch it and change it again right after.

I was reading on how pods accomplish the networking issue, and they create a lightweight "infra" container that basically does nothing but is the first container to always start and is present on the network to ensure it's available. I do not think we should use pods here as that will complicate things, but perhaps we can learn from this idea. The trickiest part would be getting the ordering right. But I do think if they are all on the same network, things get easier and I do not see a reason anymore to split them across networks.

Should I reuse the existing iop-core-network or create a new general purpose network ( if naming conventions are being looked into) for the containers

ehelms · 2026-08-10T11:38:11Z

Should I reuse the existing iop-core-network or create a new general purpose network ( if naming conventions are being looked into) for the containers

New general purpose network name that is more descriptive.

stejskalleos · 2026-08-11T08:08:22Z

+# Optional parameters - can use defaults.
+# Use 127.0.0.1: Postgres is published on IPv4 only (not ::1), and TCP
+# requires a password now that it is no longer on host networking.
+iop_fdw_database_host: "127.0.0.1"


The iop.md architecture diagram (mermaid) still draws Postgres under "Host System", outside the iop-core-network subgraph, while the prose now says Postgres is reached via postgresql:5432 on that network.

Should we update that as well?

Yeah, it's updated now

Why do some places use 127.0.0.1 and others use postgresql ?

@Alleny244 this should be addressed, as you use both in IOP.md:

flowchart LR Ansible["Ansible / tests on host"] -->|"127.0.0.1:5432 published"| PG Apps["Foreman, Pulp, IOP apps"] -->|"postgresql:5432 bridge DNS"| PG FDW["postgres_fdw inside PG"] -->|"127.0.0.1:5432 container loopback"| PG PG[(postgresql container)]

Thanks, addressed this in the docs. The difference is the network context: host-side Ansible/tests use the published 127.0.0.1:5432, while the app containers use postgresql:5432 through the bridge. The FDW runs inside the PostgreSQL container, so its 127.0.0.1 refers to the PostgreSQL container itself. I’ve added a note and linked the “PostgreSQL from three vantage points” section.

Alleny244 · 2026-08-21T08:34:05Z

Can you introduce a new document that lays out the architecture of the network? Take a look at #771 as I'd like it to go into that folder structure.

Sure I ll take a look into that and formulate one

Alleny244 · 2026-08-21T11:43:02Z

One thing that I noticed was , TLS issue affects both Candlepin and IOP: their certs are issued for candlepin / iop-core-gateway, but deploy-dev Foreman connects via localhost. Currently we are using those hostnames instead of localhost so hostname verification succeeds for production. For the host Foreman, I guess we’d also need to add both names to /etc/hosts pointing to 127.0.0.1 for TLS

ehelms · 2026-08-23T17:34:08Z

One thing that I noticed was , TLS issue affects both Candlepin and IOP: their certs are issued for candlepin / iop-core-gateway, but deploy-dev Foreman connects via localhost. Currently we are using those hostnames instead of localhost so hostname verification succeeds for production. For the host Foreman, I guess we’d also need to add both names to /etc/hosts pointing to 127.0.0.1 for TLS

For devel setup, could the certificates be issued for localhost as they are today?

Alleny244 · 2026-08-24T09:49:57Z

One thing that I noticed was , TLS issue affects both Candlepin and IOP: their certs are issued for candlepin / iop-core-gateway, but deploy-dev Foreman connects via localhost. Currently we are using those hostnames instead of localhost so hostname verification succeeds for production. For the host Foreman, I guess we’d also need to add both names to /etc/hosts pointing to 127.0.0.1 for TLS

For devel setup, could the certificates be issued for localhost as they are today?

Yeah it is covered now. The Candlepin and IOP certs include both the container names (candlepin / iop-core-gateway) and localhost as SANs. So production can connect using the container names, while devel setup can continue using localhost (no /etc/hosts changes are needed)

stejskalleos

Some things has to be addressed before we can merge.

Also, the failing CI:

TASK [pre_install : Install other dependencies] ********************************
[ERROR]: Task failed: Module failed: Depsolve Error occurred: 
 Problem: package python3.12-pyOpenSSL-26.2.0-2.el10.noarch from pulpcore requires (python3.12dist(cryptography) < 49~~ with python3.12dist(cryptography) >= 46), but none of the providers can be installed
  - package python3.12-requests-2.33.1-3.el10.noarch from pulpcore requires python3.12-pyOpenSSL >= 0.14, but none of the providers can be installed
  - cannot install both python3-cryptography-49.0.0-1.el10.x86_64 from baseos and python3-cryptography-48.0.0-3.el10.x86_64 from baseos
  - package python3-cryptography-49.0.0-1.el10.x86_64 from baseos obsoletes python3.12-cryptography < 49.0.0-1.el10 provided by python3.12-cryptography-46.0.7-3.el10.x86_64 from pulpcore
  -

do we track it somewhere? Or is there a fix / person working on it?

stejskalleos · 2026-08-27T06:17:39Z

+# Optional parameters - can use defaults.
+# Use 127.0.0.1: Postgres is published on IPv4 only (not ::1), and TCP
+# requires a password now that it is no longer on host networking.
+iop_fdw_database_host: "127.0.0.1"


@Alleny244 this should be addressed, as you use both in IOP.md:

flowchart LR Ansible["Ansible / tests on host"] -->|"127.0.0.1:5432 published"| PG Apps["Foreman, Pulp, IOP apps"] -->|"postgresql:5432 bridge DNS"| PG FDW["postgres_fdw inside PG"] -->|"127.0.0.1:5432 container loopback"| PG PG[(postgresql container)]

stejskalleos · 2026-08-27T06:28:11Z

+        "podman exec foreman curl --cacert /etc/foreman/katello-default-ca.crt"
+        "--silent --output /dev/null https://candlepin:23443/candlepin/status"


Suggested change

"podman exec foreman curl --cacert /etc/foreman/katello-default-ca.crt"

"--silent --output /dev/null https://candlepin:23443/candlepin/status"

"podman exec foreman curl --cacert /etc/foreman/katello-default-ca.crt "

"--silent --output /dev/null https://candlepin:23443/candlepin/status"

missing space, this can't work IMO

yeah, thanks, have added the space

stejskalleos · 2026-08-27T06:29:54Z

    server.run(
-        f"curl --cacert {certificates['ca_certificate']} --silent --output /dev/null "
-        f"https://localhost:23443/candlepin/status"
+        f"podman exec foreman curl --cacert {certificates['ca_certificate']} "


From AI:

certificates['ca_certificate'] resolves to a host-side path (rendered from src/vars/certificates.yml, e.g.,/var/lib/foremanctl/certs/certs/ca.crt), but this is run with podman exec foreman, i.e., inside the foreman container's filesystem, where that path doesn't exist.

The sibling test test_candlepin_status correctly uses the in-container path /etc/foreman/katello-default-ca.crt — this one should too.

Yeahh, changed!!

ehelms · 2026-08-27T14:13:29Z

+        foreman_database_host: "{{ database_host }}"
+        candlepin_database_host: "{{ database_host }}"
+        pulp_database_host: "{{ database_host }}"
+      when: database_mode == 'external'


Why is this needed here now? If the user provides the parameter this should cascade to modifying these in the src/vars/database.yml file.

yeah agreed, have moved it back to the database.yml file

ehelms · 2026-08-27T14:14:48Z

+    - name: Set external database hosts
+      ansible.builtin.set_fact:
+        pulp_database_host: "{{ database_host }}"
+      when: database_mode == 'external'


We do not support external database with the deploy-proxy action.

Have removed it

ehelms · 2026-08-27T14:16:47Z

 candlepin_ca_certificate: "{{ ca_certificate }}"
-candlepin_tomcat_key: "{{ localhost_key }}"
-candlepin_tomcat_certificate: "{{ localhost_certificate }}"
+candlepin_tomcat_key: "{{ candlepin_key }}"


Trying to remember why we set any of these certificate things here and not in src/vars/certificates.yaml.

ehelms · 2026-08-27T14:18:26Z

 localhost_client_key: "{{ certificates_ca_directory }}/private/localhost-client.key"
 localhost_client_certificate: "{{ certificates_ca_directory }}/certs/localhost-client.crt"
+candlepin_key: "{{ certificates_ca_directory }}/private/candlepin.key"
+candlepin_certificate: "{{ certificates_ca_directory }}/certs/candlepin.crt"


I think these are a bit confusing to name this way and not use tomcat in the name. Since these are for tomcat whereas Candlepin itself does get key/cert that are CAs for it's other use cases. I would align on tomcat naming. And see https://github.com/theforeman/foremanctl/pull/728/changes#r3872587738 I think we need some alignment of certs vars into the same file.

yeahh, makes sense have made the change

ehelms · 2026-08-27T14:19:31Z

 postgresql_admin_password_file: "{{ obsah_state_path }}/postgresql-admin-password"
 postgresql_admin_password: "{{ lookup('ansible.builtin.password', postgresql_admin_password_file, chars=['ascii_letters', 'digits']) }}"

-candlepin_database_host: "{{ database_host }}"


Why are we dropping these here?

Those variables used to point to localhost but with new shared network I thought of keep them under their respective roles default variables, but based on this feedback(#728 (comment)) I think it is better to have it in the initial place itself

ehelms · 2026-08-27T14:19:45Z

@@ -1,5 +1,5 @@
 ---
-database_host: localhost
+database_host: 127.0.0.1


I would have thought this would be postgresql now?

Shouldn't this stay as 127.0.0.1? database_host is used for host side tasks(login_host, pg_isready), while the containers use their own *_database_host defaults with postgresql on foreman-core-network.

That is a fair question as database_host now has two meanings with this change. I believe it should mean "this is the host that services find that database at". In an external database scenario, this should represent the host of the external database. I would turn it around then, and consider if a new variable is needed to represent the host where database tooling connects to.

Maybe like database_management_host ?

Yup, this makes sense. database_host should represent the host that the services connect to, and we can use database_management_host for host side database tooling. I ll update the pr

stejskalleos

My comments have been addressed; running foremanctl deploy works fine, and CI is green.

🍏 LGTM from my side, waiting for @ehelms for final ack

pr-processor Bot added the Not yet reviewed label Aug 5, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from c6c2b37 to de1fe85 Compare August 5, 2026 14:19

Alleny244 marked this pull request as draft August 5, 2026 14:22

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 4 times, most recently from 6bf8eb0 to a7edb41 Compare August 6, 2026 09:16

Alleny244 marked this pull request as ready for review August 6, 2026 11:47

ehelms reviewed Aug 6, 2026

View reviewed changes

Comment thread src/roles/iop_core/tasks/main.yaml Outdated

pr-processor Bot removed the Not yet reviewed label Aug 6, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 5 times, most recently from 3b4d9d7 to 6f86341 Compare August 7, 2026 11:47

stejskalleos self-requested a review August 11, 2026 07:57

stejskalleos requested changes Aug 11, 2026

View reviewed changes

pr-processor Bot added the Waiting on contributor label Aug 11, 2026

Alleny244 changed the title ~~Restrict PostgreSQL listen_addresses to localhost and IOP gateway~~ Move co-located containers onto shared foreman-core-network Aug 11, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 6f86341 to d65b220 Compare August 11, 2026 10:53

pr-processor Bot added Needs re-review and removed Waiting on contributor labels Aug 11, 2026

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 2 times, most recently from 5c5760a to 456b6e3 Compare August 11, 2026 13:16

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 0e55232 to 4c367a1 Compare August 21, 2026 11:32

Alleny244 force-pushed the fix/postgresql-listen-localhost branch 5 times, most recently from fab18cb to 8766272 Compare August 22, 2026 18:33

ehelms reviewed Aug 23, 2026

View reviewed changes

Comment thread development/playbooks/deploy-dev/deploy-dev.yaml Outdated

ehelms reviewed Aug 23, 2026

View reviewed changes

Comment thread src/roles/candlepin/defaults/main.yml Outdated

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 8766272 to 6961194 Compare August 24, 2026 09:36

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 6961194 to 142b2ad Compare August 25, 2026 10:21

stejskalleos requested changes Aug 27, 2026

View reviewed changes

pr-processor Bot added the Waiting on contributor label Aug 27, 2026

ehelms reviewed Aug 27, 2026

View reviewed changes

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 142b2ad to ecd85ee Compare August 31, 2026 07:15

pr-processor Bot added Needs re-review and removed Waiting on contributor Needs re-review labels Aug 31, 2026

stejskalleos approved these changes Aug 31, 2026

View reviewed changes

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from ecd85ee to 5ede7ef Compare September 1, 2026 06:02

Move co-located containers onto shared foreman-core-network

07b8960

Alleny244 force-pushed the fix/postgresql-listen-localhost branch from 5ede7ef to 07b8960 Compare September 1, 2026 08:25

		"podman exec foreman curl --cacert /etc/foreman/katello-default-ca.crt"
		"--silent --output /dev/null https://candlepin:23443/candlepin/status"

Uh oh!

Conversation

Alleny244 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem Statement

Changes

Tests

Checklist

Uh oh!

Uh oh!

ehelms commented Aug 7, 2026

Uh oh!

Alleny244 commented Aug 7, 2026

Uh oh!

ehelms commented Aug 7, 2026

Uh oh!

Alleny244 commented Aug 7, 2026

Uh oh!

Alleny244 commented Aug 10, 2026

Uh oh!

ehelms commented Aug 10, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Alleny244 commented Aug 21, 2026

Uh oh!

Alleny244 commented Aug 21, 2026

Uh oh!

ehelms commented Aug 23, 2026

Uh oh!

Uh oh!

Uh oh!

Alleny244 commented Aug 24, 2026

Uh oh!

stejskalleos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Alleny244 Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Alleny244 commented Aug 5, 2026 •

edited

Loading

Alleny244 Aug 31, 2026 •

edited

Loading